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SOME  CONSIDERATIONS  IN  ANALYZING  TRAINING 
COSTS  AND  JOB  PERFORMANCE 

Robert  Vineberg,  John  Joyner, 
and  Elaine  N.  Taylor 

Human  Resources  Research  Organization 
Western  Division,  Carmel,  California 


[Paper  presented  at  the  Department  of  Defense  Cost  Analysis 
Symposium,  Colorado  Springs,  Colo.,  27-29  October  1977.] 


L All  of  us  appreciate  that  huge  sums  of  money  are  spent 

I each  year  for  training.  We  are  also  aware  that  military  plan- 

[ ners  are  concerned  that  training  be  optimally  relevant  and 

t achievable  at  the  lowest  cost.  That  is  to  say,  skills  and 

i knowledges  that  pertain  to  the  actual  performance  requirements 

\ of  jobs  must  be  developed  in  future  job  holders  in  the  most 

i efficient  manner  possible.  To  achieve  cost/effectiveness,  all 

! of  the  Services  currently  emphasize  a systems  approach  to  de- 

signing and  managing  training. 

Various  models  of  military  training  systems  have  been 
'v:'  called  Systems  Approach  to  Training,  Systems  Engineering  of 

Training  and,  more  recently.  Instructional  Systems  Develop- 
ment. These  variously  named  methodologies  emphasize  front-end 
analysis,  iterative  design,  validation,  and  feedback  from  the 
field.  Each  model  for  the  development  of  training  programs 
has  been  designed  to  insure  that  the  goals  of  training  are 
achieved,  that  these  goals  are  relevant  to  the  requirements  of 
the  job,  that  they  will  be  achieved  by  a majority  of  the 
trainees,  and  that  the  costs  of  the  training  are  minimized. 

A most  distinctive  feature  of  these  various  methodologies 
is  that  they  require  analysis  of  training  content  and  outcome 
at  the  level  of  specific  tasks.  Analysis  and  measurement  are 
initiated  with  the  identification  of  a task  that  must  be  per- 
formed as  part  of  a job.  Examples  are:  cleaning  a carburetor, 
welding  a joint,  drafting  a business  letter,  or  translating  a 
Russian  newspaper  article.  To  begin  at  the  beginning,  a job 
is  identified,  tasks  within  the  job  are  specified,  training 
objectives  for  each  task  are  stated,  training  outcome  and  the 
ultimate  determination  of  job  competence  are  measured  by  the 
trainee's  (or  new  job  incumbent's)  ability  to  perform  specific 
tasks.  Most  important,  but  rarely  demonstrated,  is  that  the 
training  has  relevance  to  the  job.  Training  relevance  is 
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determined  by  how  closely  the  tasks  in  v/hich  competence  has 
been  developed  match  the  tasks  done  on  the  job. 


We  contend  that  the  use  of  the  task  as  the  unit  of  analy- 
sis and  measurement  should  be  extended  to  cost/effectiveness 
studies  of  training.  Measuring  performance  at  the  level  of 
the  task  will  provide  a more  sensitive  means  for  detecting 
training  inefficiency.  Definitive  comparisons  can  be  made 
among  students  from  the  same  training  program  and  even  from 
different  programs.  Comparisons  can  also  be  made  between  the 
performance  of  graduates  and  the  frequency  of  assignment  to 
the  "trained"  tasks  on  the  job.  Comparisons  between  perfor- 
mance and  requirements  can  thus  be  used  to  identify  under- 
training and  overtraining. 

In  the  past,  studies  of  the  effectiveness  of  training 
have  tended  to  use  measures  of  output  that  were  of  a global 
nature.  For  example,  supervisors  filled  out  rating  forms 
with  such  items  as  "professional  performance"  and  "military 
appearance."  Such  indices  of  output  mask  or  are  irrelevant 
to  effects  at  the  task  level. 

More  recently,  training  has  been  evaluated  by  measuring 
outcomes  that  were  intermediate  in  nature — for  example, 
counting  the  number  of  graduates  per  unit  of  time.  Descrip- 
tions of  this  kind  assume  that  different  varieties  of  train- 
ing are  essentially  equivalent  and  product  equivalent  pro- 
ducts. To  evaluate  training  in  terms  of  the  number  of 
trainees  who  are  performing  in  nominally  similar  jobs  is  to 
assume  that  alternate  training  programs  are  equally  appropri- 
ate to  different  duty  positions  or  billets.  These  assump- 
tions disregard  effectiveness  and  measure  instead  the  effi- 
ciency of  the  delivery  system. 

Most  approaches  to  the  evaluation  of  training  effective- 
ness fail  to  go  beyond  the  training  process  itself.  They  do 
not  require  or  even  imply  that  performance  on  the  job  should 
be  measured  by  the  retention  and  application  of  skill  and 
knowledge  in  different  job  settings.  Most  studies  focus  on 
what  is  learned  in  the  most  immediate  sense;  for  example, 
whether  exposure  to  a particular  audio-visual  device  or  simu- 
lator results  in  more  effective  learning  as  evidenced  by 
course  grade.  The  correlation  between  grades  and  job  perfor- 
mance is,  of  course,  at  best  imperfect.  This  is  perfectly 
true  when  grades  are  conditioned,  as  they  so  often  are,  by 
verbal  skills  and  the  ability  to  do  well  on  paper-and-pencil 
tests  of  information. 

To  assess  training  output  adequately,  we  need  to  direct 
our  attention  to  the  measurement  of  task  performance  at  least 
at  the  completion  of  training,  or  better  still,  after  arrival 
on  the  job.  Task-level  performance  tests  have  been  used  in 
the  past  mostly  for  research  purposes.  However,  the  Army,  in 
the  Skill  Qualification  Testing  program,  has  been  developing 
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System  for  at  least  two  years.  Where  the  use  of  task-level 
performance  tests  is  either  inappropriate  or  not  possible  for 
cost  or  other  reasons,  performance-based  written  tests  for 
tasks  are  being  developed.  Also,  a supervisor  rating  proce- 
dure for  evaluating  the  performance  of  specific  tasks  is  being 
developed  in  the  so-called  Performance  Certification  component 
of  Skill  Qualification  Tests. 

Supervisor  ratings  may  yet  develop  into  the  best  method 
for  evaluating  the  cost-effectiveness  of  training  programs. 
Ratings  of  job/task  performance  are  clearly  the  most  feasible 
means  for  assessing  job  performance.  In  a study  for  the  Of- 
fice of  Naval  Research,  we  recently  developed  scales  for  rat- 
ing performance  at  the  task  level  in  ten  Navy  jobs.  Items  were 
derived  using  data  from  the  Navy  Occupational  Task  Analysis 
Program  (NOTAP)  and  from  an  analysis  of  these  same  jobs  using 
McCormick's  Position  Analysis  Questionnaire  (PAQ) . Supervi- 
sors evaluated  performance  in  from  40  to  93  tasks  based  on 
NOTAP  job  descriptions  and  from  34  to  56  items  based  on  the  PAQ 
job  analysis.*  The  supervisors  were  able  to  discriminate  dif- 
ferences in  job  performance  of  their  assigned  subordinates  at 
this  level  of  analysis. 

Mn  summary,  a task-level  approach  to  cost-effectiveness 
analysis  would  use  information  from  an  analysis  of  training 
content,  from  occupational  analysis,  and  from  performance  rat- 
ings. It  would  relate  the  costs  of  training  and  number  of 
tasks  in  which  training  is  provided  to  the  number  of  tasks  in 
which  proficiency  is  attained  at  the  completion  of  training. 
Thus,  the  effectiveness  of  a training  program  might  be  expres- 
sed by  the  number  of  tasks  taught,  times  the  quality  of  per- 
formance. Similarly,  training  costs  could  also  be  related  to 
the  actual  frequency  and  quality  of  task  performance  in  dif- 
ferent types  of  jobs. 


Differences  in  the  number  of  items  administered  were 
based  on  the  Mavy  job  and  pay  grade  of  the  incumbent  being 
rated. 
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